Method and system for fusing image data

ABSTRACT

A method and system for fusing image data. The method may include obtaining a first volume image and a second volume image. The method may further include casting a plurality of rays through at least one of the first volume image or the second volume image. Each of the plurality of rays may correspond to a pixel of an image to be displayed. For each of at least a portion of the plurality of rays, the at least one processor may further be directed to cause the system to set a series of sampling positions along the ray. The method may further include selecting a reference position from the series of sampling positions. The method may further include determining fusion data of the ray. The method may further include determining a pixel value of a pixel of the image to be displayed that corresponds to the ray.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority of Chinese Patent Application No.201810677958.2 filed on Jun. 27, 2018, the contents of which are herebyincorporated by reference.

TECHNICAL FIELD

The present disclosure generally relates to a method and system forimage processing, and more particularly to systems and methods forfusing image data.

BACKGROUND

In clinical applications, medical imaging plays an impotent role forproviding anatomical information of a patient. However, the informationprovided by single-modality medical images (e.g., a computed tomography(CT) image, a positron emission computed tomography (PET) images, amagnetic resonance (MR) image), may be insufficient for an intendedpurpose, e.g., diagnosis, treatment planning or execution. In such asituation, a fused medical image of different modalities may providemore accurate and comprehensive anatomical and/or functional informationof, e.g., a lesion, thereby facilitating a doctor to make an accuratediagnosis, to develop a suitable treatment plan, and/or to execute atreatment plan.

An image fusion is proposed in the art for generating a multi-modalityimage based on multiple single-modality medical images, which mayinclude obtaining single-modality images of a same target that areacquired by imaging devices of a same or different sources (e.g., a sameor different modalities), and processing the multiple single-modalityimages to extract useful information therefrom and integrating theextracted information into a same image for viewing and/or furtherprocessing. A fused mono-modality or multi-modality image may have asuperior performance over the single-modality images on the basis ofwhich the fused image is generated. According to the imaging modalitiesof images subjected to the image fusion, approaches for performing animage fusion may include, e.g., a mono-modality image fusion approachand a multi-modality image fusion approach. A mono-modality image fusionapproach may refer to a fusion of images of the same imaging modality,such as an image fusion of single-photon emission computed tomography(SPECT) images, an image fusion of MR images, etc. A multi-modalityimage fusion approach may refer to an image fusion performed on imagesof different imaging modalities, such as an image fusion of a SPECTimage and an MR image, an image fusion of a PET image and a CT image.

In the art, an image fusion performed on a PET image obtained via amaximum intensity projection (MIP) and a CT image obtained via a volumerendering (VolRen) is generally performed by fusing each two-dimensionalMIP image of the PET image (including a plurality of the two dimensionalMIP image) and the corresponding two-dimensional VolRen image of the CTimage (including a plurality of two-dimensional VolRen images) in animage-by-image manner. However, via such an approach, it is difficult toaccurately represent depth information of tissue in the MIP image, andthe obtained fused image may represent the scanned portion of thepatient inaccurately. As a result, the analysis and diagnosis based onthe fused image may be negatively influenced.

Therefore, it is desirable to provide a method and system for generatinga fusion image (or referred to as a fused image) providing depthinformation.

SUMMARY

According to an aspect of the present disclosure, a system is provided.The system may include at least one storage medium storing a set ofinstructions and at least one processor configured to communicate withthe at least one storage medium. When executing the set of instructions,the at least one processor may be directed to cause the system to obtaina first volume image and a second volume image. The first volume imagemay include a plurality of first voxels. The second volume image mayinclude a plurality of second voxels. The at least one processor mayalso be directed to cause the system to cast a plurality of rays throughat least one of the first volume image or the second volume image. Eachof the plurality of rays may correspond to a pixel of an image to bedisplayed. For each of at least a portion of the plurality of rays, theat least one processor may further be directed to cause the system toset a series of sampling positions along the ray. The at least oneprocessor may further be directed to cause the system to select, basedon voxel values of the first voxels along the ray, a reference positionfrom the series of sampling positions. The at least one processor mayfurther be directed to cause the system to determine, based on a voxelvalue of at least one first voxel corresponding to the referenceposition and voxel values of second voxels corresponding to at leastsome of the series of sampling positions, fusion data of the ray. The atleast one processor may further be directed to cause the system todetermine, based at least in part on the fusion data of the ray, a pixelvalue of a pixel of the image to be displayed that corresponds to theray.

In some embodiments, to select, based on voxel values of the firstvoxels along the ray, a reference position from the series of samplingpositions, the at least one processor may further be directed to causethe system to determine a reference voxel. The reference voxel may bethe first voxel that has a highest voxel value among the first voxelsalong the ray. The at least one processor may further be directed tocause the system to designate the sampling position corresponding to thereference voxel as the reference position.

In some embodiments, the series of sampling positions may include astart sampling position and an end sampling position. To determine,based on a voxel value of at least one first voxel corresponding to thereference position and voxel values of second voxels corresponding to atleast some of the series of sampling positions, fusion data of the ray,the at least one processor may further be directed to cause the systemto determine, based on the voxel value of the at least one first voxelcorresponding to the reference position and the voxel values of thesecond voxels corresponding to sampling positions from the startsampling position to the end sampling position, the fusion data of theray.

In some embodiments, to determine, based on the voxel value of the atleast one first voxel corresponding to the reference position and thevoxel values of the second voxels corresponding to sampling positionsfrom the start sampling position to the end sampling position, thefusion data of the ray, the at least one processor may further directedto cause the system to obtain, at the reference position, a firstsampling value based on the voxel value of the at least one first voxelcorresponding to the reference position. The at least one processor mayfurther directed to cause the system to obtain, at each samplingposition from the start sampling position to the end sampling position,a second sampling value based on the voxel value of at least one secondvoxel corresponding to the sampling position, thereby obtaining aplurality of second sampling values. The at least one processor mayfurther directed to cause the system to determine, based on the firstsampling value and the plurality of second sampling values, the fusiondata of the ray.

In some embodiments, the first sampling value may be obtained byperforming an interpolation or extrapolation on the voxel value of theat least one first voxel corresponding to the reference position. Thesecond sampling value may be obtained by performing an interpolation orextrapolation on the voxel value of the at least one second voxelcorresponding to the sampling position.

In some embodiments, to determine, based on the first sampling value andthe plurality of second sampling values, the fusion data of the ray, theat least one processor may further be directed to cause the system todesignate the second sampling value of the start sampling position as afirst preliminary value. The at least one processor may further bedirected to cause the system to obtain a second preliminary value byupdating, for each sampling position from the sampling position next tothe start sampling position to the reference position and based at leastin part on the second sampling value of the sampling position, the firstpreliminary value. The at least one processor may further be directed tocause the system to obtain a third preliminary value by updating thesecond preliminary value based on the first sampling value. The at leastone processor may further be directed to cause the system to obtain thefusion data of the ray by updating, for each sampling position from thesampling position next to the reference position to the end samplingposition and based at least in part on the second sampling value of thesampling position, the third preliminary value.

In some embodiments, the obtaining a second preliminary value byupdating, for each sampling position from the sampling position next tothe start sampling position to the reference position and based at leastin part on the second sampling value of the sampling position, the firstpreliminary value may include combining a current first preliminaryvalue and the second sampling value using an alpha blending technique.The obtaining a third preliminary value by updating the secondpreliminary value based on the first sampling value may includecombining the second preliminary value and the first sampling valueusing the alpha blending technique. The obtaining the fusion data of theray by updating, for each sampling position from the sampling positionnext to the reference position to the end sampling position and based atleast in part on the second sampling value of the sampling position, thethird preliminary value may include combining a current thirdpreliminary value and the second sampling value using the alpha blendingtechnique.

In some embodiments, to obtain a first volume image and a second volumeimage, the at least one processor may be further directed to cause thesystem to perform an image registration between the first volume imageand the second volume image, wherein the first volume image and thesecond volume image include image regions corresponding to a sameobject.

In some embodiments, to determine, based at least in part on the fusiondata of the ray, a pixel value of a pixel of the image to be displayedthat corresponds to the ray may be based on a volume renderingtechnique.

In some embodiments, to determine, based at least in part on the fusiondata of the ray, a pixel value of a pixel of the image to be displayedthat corresponds to the ray, the at least one processor may be directedto cause the system to perform a window width adjustment or a windowlevel adjustment on the fusion data of the ray.

In some embodiments, the first volume image may be obtained via amaximum intensity projection (MIP) technique.

In some embodiments, the first volume image may be a positron emissioncomputed tomography (PET) image. The second volume image may be acomputed tomography (CT) image or a magnetic resonance (MR) image.

According to another aspect of the present disclosure, a method isprovided. The method may include obtaining a first volume image and asecond volume image. The first volume image may include a plurality offirst voxels. The second volume image may include a plurality of secondvoxels. The method may further include casting a plurality of raysthrough at least one of the first volume image or the second volumeimage. Each of the plurality of rays may correspond to a pixel of animage to be displayed. For each of at least a portion of the pluralityof rays, the at least one processor may further be directed to cause thesystem to set a series of sampling positions along the ray. The methodmay further include selecting, based on voxel values of the first voxelsalong the ray, a reference position from the series of samplingpositions. The method may further include determining, based on a voxelvalue of at least one first voxel corresponding to the referenceposition and voxel values of second voxels corresponding to at leastsome of the series of sampling positions, fusion data of the ray. Themethod may further include determining, based at least in part on thefusion data of the ray, a pixel value of a pixel of the image to bedisplayed that corresponds to the ray.

According to still a further aspect of the present disclosure, anon-transitory computer readable medium is provided. The non-transitorycomputer readable medium storing instructions, the instructions, whenexecuted by a computer, may cause the computer to implement a method.The method may include one or more of the following operations. Themethod may include obtaining a first volume image and a second volumeimage. The first volume image may include a plurality of first voxels.The second volume image may include a plurality of second voxels. Themethod may further include casting a plurality of rays through at leastone of the first volume image or the second volume image. Each of theplurality of rays may correspond to a pixel of an image to be displayed.For each of at least a portion of the plurality of rays, the at leastone processor may further be directed to cause the system to set aseries of sampling positions along the ray. The method may furtherinclude selecting, based on voxel values of the first voxels along theray, a reference position from the series of sampling positions. Themethod may further include determining, based on a voxel value of atleast one first voxel corresponding to the reference position and voxelvalues of second voxels corresponding to at least some of the series ofsampling positions, fusion data of the ray. The method may furtherinclude determining, based at least in part on the fusion data of theray, a pixel value of a pixel of the image to be displayed thatcorresponds to the ray.

Additional features will be set forth in part in the description whichfollows, and in part will become apparent to those skilled in the artupon examination of the following and the accompanying drawings or maybe learned by production or operation of the examples. The features ofthe present disclosure may be realized and attained by practice or useof various aspects of the methodologies, instrumentalities andcombinations set forth in the detailed examples discussed below.

BRIEF DESCRIPTIONS OF THE DRAWINGS

The present disclosure is further described in terms of exemplaryembodiments. These exemplary embodiments are described in detail withreference to the drawings. These embodiments are non-limiting exemplaryembodiments, in which like reference numerals represent similarstructures throughout the several views of the drawings, and wherein:

FIG. 1 is a schematic diagram illustrating an exemplary system accordingto some embodiments of the present disclosure;

FIG. 2 is a schematic diagram illustrating exemplary hardware and/orsoftware components of an exemplary computing device according to someembodiments of the present disclosure;

FIG. 3 is a schematic diagram illustrating exemplary hardware and/orsoftware components of an exemplary mobile device according to someembodiments of the present disclosure;

FIG. 4 is a block diagram illustrating an exemplary processing device140 according to some embodiments of the present disclosure;

FIG. 5 is a block diagram illustrating an exemplary ray casting module420 according to some embodiments of the present disclosure;

FIG. 6 is a flowchart illustrating an exemplary process for determininga fusion result of two volume image according to some embodiments of thepresent disclosure;

FIG. 7 is a flowchart illustrating an exemplary process for determininga fusion data of a ray according to some embodiments of the presentdisclosure;

FIG. 8 is a flowchart illustrating an exemplary process for image fusionaccording to some embodiments of the present disclosure;

FIG. 9 is a flowchart illustrating another exemplary process for imagefusion according to some embodiments of the present disclosure;

FIG. 10 is a block diagram illustrating an exemplary image fusion systemaccording to some embodiments of the present disclosure;

FIG. 11 is a block diagram illustrating an exemplary medical equipmentaccording to some embodiments of the present disclosure

FIG. 12a and FIG. 12b are fusion images generated based on volumerendering and MIP techniques according to some embodiments of thepresent disclosure;

FIG. 13a and FIG. 13b are fusion images with different window widthsand/or window levels according to some embodiments of the presentdisclosure; and

FIG. 14 is a fusion image according to some embodiments of the presentdisclosure.

DETAILED DESCRIPTION

The present disclosure is directed to a method and system for performingan image fusion of different images to obtain a fused image. Forexample, the fused image may be used for a treatment of a disease, adiagnosis of a disease, a synchronous motion control, a research, etc.In some embodiments, the image fusion may be performed for generating afused medical image to provide more accurate anatomical information of apatient. For example, the image fusion may be performed on a firstvolume image (e.g., a PET image via an MIP approach) and a second volumeimage (e.g., a CT image or an MR image). During the image fusion, aplurality of rays may be casted through at least one of the first volumeimage or the second volume image. Each of the plurality of rays maycorrespond to a pixel of a fused image. Fusion data may be obtained foreach of the plurality of rays, and based on the fusion data, a pixelvalue of a corresponding pixel (e.g., the color of a pixel, the grayvalue of a pixel) of the fused image may be determined. For each of theplurality of rays, voxels of the first volume image and voxels of thesecond volume image that are along the ray may be selectively used toobtain the corresponding fusion data, so as to improve the accuracy ofthe depth information in the fused image. The fused image with improveddepth information may simplify, and/or improve the efficiency and/or theaccuracy of a diagnosis performed based on the fused image.

The following description is presented to enable any person skilled inthe art to make and use the present disclosure, and is provided in thecontext of a particular application and its requirements. Variousmodifications to the disclosed embodiments will be readily apparent tothose skilled in the art, and the general principles defined herein maybe applied to other embodiments and applications without departing fromthe spirit and scope of the present disclosure. Thus, the presentdisclosure is not limited to the embodiments shown, but is to beaccorded the widest scope consistent with the claims.

The flowcharts used in the present disclosure illustrate operations thatsystems implement according to some embodiments of the presentdisclosure. It is to be expressly understood, the operations of theflowcharts may be implemented not in order. Conversely, the operationsmay be implemented in inverted order, or simultaneously. Moreover, oneor more other operations may be added to the flowcharts. One or moreoperations may be removed from the flowcharts.

In the following detailed description, numerous specific details are setforth by way of examples in order to provide a thorough understanding ofthe relevant disclosure. However, it should be apparent to those skilledin the art that the present disclosure may be practiced without suchdetails. In other instances, well known methods, procedures, systems,components, and/or circuitry have been described at a relativelyhigh-level, without detail, in order to avoid unnecessarily obscuringaspects of the present disclosure. Various modifications to thedisclosed embodiments will be readily apparent to those skilled in theart, and the general principles defined herein may be applied to otherembodiments and applications without departing from the spirit and scopeof the present disclosure. Thus, the present disclosure is not limitedto the embodiments shown, but to be accorded the widest scope consistentwith the claims.

The terminology used herein is for the purpose of describing particularexample embodiments only and is not intended to be limiting. As usedherein, the singular forms “a,” “an,” and “the,” may be intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprise”,“comprises”, and/or “comprising”, “include”, “includes”, and/or“including”, when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof.

It will be understood that the term “system,” “unit,” “module,” and/or“block” used herein are one method to distinguish different components,elements, parts, section or assembly of different level in ascendingorder. However, the terms may be displaced by another expression if theyachieve the same purpose.

Generally, the word “module,” “sub-module,” “unit,” or “block,” as usedherein, refers to logic embodied in hardware or firmware, or to acollection of software instructions. A module, a unit, or a blockdescribed herein may be implemented as software and/or hardware and maybe stored in any type of non-transitory computer-readable medium oranother storage device. In some embodiments, a softwaremodule/unit/block may be compiled and linked into an executable program.It will be appreciated that software modules can be callable from othermodules/units/blocks or from themselves, and/or may be invoked inresponse to detected events or interrupts.

Software modules/units/blocks configured for execution on computingdevices (e.g., processor 210 as illustrated in FIG. 2) may be providedon a computer-readable medium, such as a compact disc, a digital videodisc, a flash drive, a magnetic disc, or any other tangible medium, oras a digital download (and can be originally stored in a compressed orinstallable format that needs installation, decompression, or decryptionprior to execution). Such software code may be stored, partially orfully, on a storage device of the executing computing device, forexecution by the computing device. Software instructions may be embeddedin a firmware, such as an EPROM. It will be further appreciated thathardware modules/units/blocks may be included in connected logiccomponents, such as gates and flip-flops, and/or can be included ofprogrammable units, such as programmable gate arrays or processors. Themodules/units/blocks or computing device functionality described hereinmay be implemented as software modules/units/blocks, but may berepresented in hardware or firmware. In general, themodules/units/blocks described herein refer to logicalmodules/units/blocks that may be combined with othermodules/units/blocks or divided into sub-modules/sub-units/sub-blocksdespite their physical organization or storage. The description may beapplicable to a system, an engine, or a portion thereof.

It will be understood that when a unit, engine, module or block isreferred to as being “on,” “connected to,” or “coupled to,” anotherunit, engine, module, or block, it may be directly on, connected orcoupled to, or communicate with the other unit, engine, module, orblock, or an intervening unit, engine, module, or block may be present,unless the context clearly indicates otherwise. As used herein, the term“and/or” includes any and all combinations of one or more of theassociated listed items.

These and other features, and characteristics of the present disclosure,as well as the methods of operation and functions of the relatedelements of structure and the combination of parts and economies ofmanufacture, may become more apparent upon consideration of thefollowing description with reference to the accompanying drawings, allof which form a part of this disclosure. It is to be expresslyunderstood, however, that the drawings are for the purpose ofillustration and description only and are not intended to limit thescope of the present disclosure.

FIG. 1 is a schematic diagram illustrating an exemplary system accordingto some embodiments of the present disclosure. As shown in FIG. 1, thesystem 100 may include an apparatus 110, a network 120, one or moreterminals 130, a processing device 140, and a storage device 150. Thecomponents in the system 100 may be connected in various ways. In someembodiments, the apparatus 110 may be connected to the processing device140 through the network 120. In some embodiments, the apparatus 110 maybe connected to the processing device 140 directly as indicated by thebi-directional broken arrow 161. In some embodiments, the storage device150 may be connected to the processing device 140 directly or throughthe network 120. In some embodiments, the terminal 130 may be connectedto the processing device 140 directly as indicated by the bi-directionalbroken arrow 162 or through the network 120.

In some embodiments, the apparatus 110 may be an RT device. In someembodiments, the RT device may deliver a radiation beam to an object(e.g., a patient, or a phantom) or a portion thereof. In someembodiments, the RT device may include a linear accelerator (alsoreferred to as “linac”). The linac may generate and emit a radiationbeam (e.g., an X-ray beam) from a treatment head. The radiation beam maypass through one or more collimators (e.g., a multi-leaf collimator(MLC)) of certain shapes, and enter into the object. In someembodiments, the radiation beam may include electrons, photons, or othertypes of radiation. In some embodiments, the energy of the radiationbeam may be in the megavoltage range (e.g., >1 MeV), and may thereforebe referred to as a megavoltage beam. The treatment head may be coupledto a gantry. The gantry may rotate, for example, clockwise orcounter-clockwise around a gantry rotation axis. In some embodiments,the treatment head may rotate along with the gantry. In someembodiments, the RT device may include a table configured to support theobject during radiation treatment.

In some embodiments, the apparatus 110 may be an imaging device. Theimaging device may generate or provide image(s) by scanning an object ora part thereof. In some embodiments, the imaging device may be a medicalimaging device, for example, a positron emission tomography (PET)device, a single-photon emission computed tomography (SPECT) device, acomputed tomography (CT) device, a magnetic resonance imaging (MRI)device, an ultrasonography device, an X-ray photography device, or thelike, or any combination thereof. In some embodiments, the imagingdevice may include a gantry to support one or more imaging componentsconfigured to imaging the object, and/or a table configured to supportthe object during an imaging process. In some embodiments, the imagingdevice may include a single-modality scanner. The single-modalityscanner may include an MRI scanner, a CT scanner, a PET scanner, or thelike, or any combination thereof. In some embodiments, the imagingdevice may include a multi-modality scanner. The multi-modality scannermay include a positron emission tomography-computed tomography (PET-CT)scanner, a positron emission tomography-magnetic resonance imaging(PET-MRI) scanner, or the like, or any combination thereof. In someembodiments, the imaging device may transmit the image(s) via thenetwork 120 to the processing device 140, the storage device 150, and/orthe terminal(s) 130. For example, the image(s) may be sent to theprocessing device 140 for further processing or may be stored in thestorage device 150.

In some embodiments, the apparatus 110 may be an integrated device of animaging device and an RT device. In some embodiments, the apparatus 110may include one or more surgical instruments. In some embodiments, theapparatus 110 may include an operating table (or table for brevity)configured to support an object during surgery. The table may support anobject during a treatment process or imaging process, and/or support aphantom during a correction process of the apparatus 110. The table maybe adjustable and/or movable to suit for different applicationscenarios.

In some embodiments, the object to be treated or scanned (also referredto as imaged) may include a body, substance, or the like, or anycombination thereof. In some embodiments, the object may include aspecific portion of a body, such as a head, a thorax, an abdomen, or thelike, or any combination thereof. In some embodiments, the object mayinclude a specific organ, such as a breast, an esophagus, a trachea, abronchus, a stomach, a gallbladder, a small intestine, a colon, abladder, a ureter, a uterus, a fallopian tube, etc. In the presentdisclosure, “object” and “subject” are used interchangeably.

The network 120 may include any suitable network that can facilitateexchange of information and/or data for the system 100. In someembodiments, one or more components of the system 100 (e.g., theapparatus 110, the terminal 130, the processing device 140, the storagedevice 150) may communicate information and/or data with one or moreother components of the system 100 via the network 120. For example, theprocessing device 140 may obtain one or more instructions from theterminal 130 via the network 120. As another example, the processingdevice 140 may obtain one or more images and/or image data (e.g., volumeimages and/or volume image data) from the apparatus 110 or the storagedevice 150 via the network 120. The network 120 may be and/or include apublic network (e.g., the Internet), a private network (e.g., a localarea network (LAN), a wide area network (WAN)), etc.), a wired network(e.g., an Ethernet network), a wireless network (e.g., an 802.11network, a Wi-Fi network, etc.), a cellular network (e.g., a Long TermEvolution (LTE) network), a frame relay network, a virtual privatenetwork (“VPN”), a satellite network, a telephone network, routers,hubs, switches, server computers, and/or any combination thereof. Merelyby way of example, the network 120 may include a cable network, awireline network, a fiber-optic network, a telecommunications network,an intranet, a wireless local area network (WLAN), a metropolitan areanetwork (MAN), a public telephone switched network (PSTN), a Bluetooth™network, a ZigBee™ network, a near field communication (NFC) network, orthe like, or any combination thereof. In some embodiments, the network120 may include one or more network access points. For example, thenetwork 120 may include wired and/or wireless network access points suchas base stations and/or internet exchange points through which one ormore components of the system 100 may be connected to the network 120 toexchange data and/or information.

The terminal(s) 130 may enable interactions between a user and thesystem 100. The terminal(s) 130 may include a mobile device 130-1, atablet computer 130-2, a laptop computer 130-3, or the like, or anycombination thereof. In some embodiments, the mobile device 130-1 mayinclude a smart home device, a wearable device, a mobile device, avirtual reality device, an augmented reality device, or the like, or anycombination thereof. Merely by way of example, the terminal 130 mayinclude a mobile device as illustrated in FIG. 3. In some embodiments,the smart home device may include a smart lighting device, a controldevice of an intelligent electrical apparatus, a smart monitoringdevice, a smart television, a smart video camera, an interphone, or thelike, or any combination thereof. In some embodiments, the wearabledevice may include a bracelet, a footgear, eyeglasses, a helmet, awatch, clothing, a backpack, a smart accessory, or the like, or anycombination thereof. In some embodiments, the mobile device may includea mobile phone, a personal digital assistant (PDA), a gaming device, anavigation device, a point of sale (POS) device, a laptop, a tabletcomputer, a desktop, or the like, or any combination thereof. In someembodiments, the virtual reality device and/or the augmented realitydevice may include a virtual reality helmet, virtual reality glasses, avirtual reality patch, an augmented reality helmet, augmented realityglasses, an augmented reality patch, or the like, or any combinationthereof. For example, the virtual reality device and/or the augmentedreality device may include a Google Glass™, an Oculus Rift™, aHololens™, a Gear VR™, etc. In some embodiments, the terminal(s) 130 maybe part of the processing device 140.

The processing device 140 may process data and/or information obtainedfrom the apparatus 110, the terminal 130, and/or the storage device 150.For example, the processing device 140 may obtain one or more volumeimage and/or volume image data. As another example, the processingdevice 140 may obtain a registration result by registering the one ormore volume image and/or volume image data. As still another example,the processing device 140 may cast a plurality of rays through the oneor more volume image and/or volume image data to obtain fusion data. Asa further example, the processing device 140 may generate an image to bedisplayed based on the fusion data

In some embodiments, the processing device 140 may be a computer, a userconsole, a single server or a server group, etc. The server group may becentralized or distributed. In some embodiments, the processing device140 may be local or remote. For example, the processing device 140 mayaccess information and/or data stored in the apparatus 110, the terminal130, and/or the storage device 150 via the network 120. As anotherexample, the processing device 140 may be directly connected to theapparatus 110, the terminal 130, and/or the storage device 150 to accessstored information and/or data. In some embodiments, the processingdevice 140 may be implemented on a cloud platform. Merely by way ofexample, the cloud platform may include a private cloud, a public cloud,a hybrid cloud, a community cloud, a distributed cloud, an inter-cloud,a multi-cloud, or the like, or any combination thereof. In someembodiments, the processing device 140 may be implemented by a computingdevice 200 having one or more components as illustrated in FIG. 2.

The storage device 150 may store data, instructions, and/or any otherinformation. In some embodiments, the storage device 150 may store dataobtained from the apparatus 110, the terminal 130 and/or the processingdevice 140. In some embodiments, the storage device 150 may store dataand/or instructions that the processing device 140 may execute or use toperform exemplary methods described in the present disclosure. In someembodiments, the storage device 150 may include a mass storage device, aremovable storage device, a volatile read-and-write memory, a read-onlymemory (ROM), or the like, or any combination thereof. Exemplary massstorage may include a magnetic disk, an optical disk, a solid-statedrive, etc. Exemplary removable storage may include a flash drive, afloppy disk, an optical disk, a memory card, a zip disk, a magnetictape, etc. Exemplary volatile read-and-write memory may include a randomaccess memory (RAM). Exemplary RAM may include a dynamic RAM (DRAM), adouble date rate synchronous dynamic RAM (DDR SDRAM), a static RAM(SRAM), a thyristor RAM (T-RAM), and a zero-capacitor RAM (Z-RAM), etc.Exemplary ROM may include a mask ROM (MROM), a programmable ROM (PROM),an erasable programmable ROM (EPROM), an electrically erasableprogrammable ROM (EEPROM), a compact disk ROM (CD-ROM), and a digitalversatile disk ROM, etc. In some embodiments, the storage device 150 maybe implemented on a cloud platform. Merely by way of example, the cloudplatform may include a private cloud, a public cloud, a hybrid cloud, acommunity cloud, a distributed cloud, an inter-cloud, a multi-cloud, orthe like, or any combination thereof.

In some embodiments, the storage device 150 may be connected to thenetwork 120 to communicate with one or more other components in thesystem 100 (e.g., the processing device 140, the terminal 130, etc.).One or more components in the system 100 may access the data orinstructions stored in the storage device 150 via the network 120. Insome embodiments, the storage device 150 may be directly connected to orcommunicate with one or more other components in the system 100 (e.g.,the processing device 140, the terminal 130, etc.). In some embodiments,the storage device 150 may be part of the processing device 140. In someembodiments, the processing device 140 may be connected to orcommunicate with the apparatus 110 via the network 120, or at thebackend of the processing device 140.

FIG. 2 is a schematic diagram illustrating exemplary hardware and/orsoftware components of an exemplary computing device on which theprocessing device 140 may be implemented according to some embodimentsof the present disclosure. As illustrated in FIG. 2, the computingdevice 200 may include a processor 210, a storage 220, an input/output(I/O) 230, and a communication port 240.

The processor 210 may execute computer instructions (e.g., program code)and perform functions of the processing device 140 in accordance withtechniques described herein. The computer instructions may include, forexample, routines, programs, objects, components, data structures,procedures, modules, and functions, which perform particular functionsdescribed herein. For example, the processor 210 may process image(s)obtained from the apparatus 110, the terminal 130, the storage device150, and/or any other component of the system 100. In some embodiments,the processor 210 may process the image(s) based on information relatingto a treatment plan. The treatment plan may be obtained from a treatmentplanning system (TPS) associated with the system 100. The informationrelating to the treatment plan may include preoperative medical image(s)representing the internal anatomical information of an object to betreated or imaged. In some embodiments, the processor 210 may generateaugmented reality (AR) image(s) based on the image(s) or informationobtained from the terminal 130, the storage device 150, and/or any othercomponent of the system 100. The AR image(s) may represent the externalsurface information of the object and/or the internal anatomicalinformation of the object. In some embodiments, the processor 210 mayinclude one or more hardware processors, such as a microcontroller, amicroprocessor, a reduced instruction set computer (RISC), anapplication specific integrated circuits (ASICs), anapplication-specific instruction-set processor (ASIP), a centralprocessing unit (CPU), a graphics processing unit (GPU), a physicsprocessing unit (PPU), a microcontroller unit, a digital signalprocessor (DSP), a field programmable gate array (FPGA), an advancedRISC machine (ARM), a programmable logic device (PLD), any circuit orprocessor capable of executing one or more functions, or the like, orany combinations thereof.

Merely for illustration, only one processor is described in thecomputing device 200. However, it should be noted that the computingdevice 200 in the present disclosure may also include multipleprocessors, thus operations and/or method steps that are performed byone processor as described in the present disclosure may also be jointlyor separately performed by the multiple processors. For example, if inthe present disclosure the processor of the computing device 200executes both operation A and operation B, it should be understood thatoperation A and operation B may also be performed by two or moredifferent processors jointly or separately in the computing device 200(e.g., a first processor executes operation A and a second processorexecutes operation B, or the first and second processors jointly executeoperations A and B).

The storage 220 may store data/information obtained from the apparatus110, the terminal 130, the storage device 150, and/or any othercomponent of the system 100. In some embodiments, the storage 220 mayinclude a mass storage device, a removable storage device, a volatileread-and-write memory, a read-only memory (ROM), or the like, or anycombination thereof. For example, the mass storage may include amagnetic disk, an optical disk, a solid-state drive, etc. The removablestorage may include a flash drive, a floppy disk, an optical disk, amemory card, a zip disk, a magnetic tape, etc. The volatileread-and-write memory may include a random access memory (RAM). The RAMmay include a dynamic RAM (DRAM), a double date rate synchronous dynamicRAM (DDR SDRAM), a static RAM (SRAM), a thyristor RAM (T-RAM), and azero-capacitor RAM (Z-RAM), etc. The ROM may include a mask ROM (MROM),a programmable ROM (PROM), an erasable programmable ROM (EPROM), anelectrically erasable programmable ROM (EEPROM), a compact disk ROM(CD-ROM), and a digital versatile disk ROM, etc. In some embodiments,the storage 220 may store one or more programs and/or instructions toperform exemplary methods described in the present disclosure. Forexample, the storage 220 may store a program for fusing images.

The I/O 230 may input and/or output signals, data, information, etc. Insome embodiments, the I/O 230 may enable a user interaction with theprocessing device 140. In some embodiments, the I/O 230 may include aninput device and an output device. Examples of the input device mayinclude a keyboard, a mouse, a touch screen, a microphone, or the like,or a combination thereof. Examples of the output device may include adisplay device, a loudspeaker, a printer, a projector, or the like, or acombination thereof. Examples of the display device may include a liquidcrystal display (LCD), a light-emitting diode (LED)-based display, aflat panel display, a curved screen, a television device, a cathode raytube (CRT), a touch screen, or the like, or a combination thereof.

The communication port 240 may be connected to a network (e.g., thenetwork 120) to facilitate data communications. The communication port240 may establish connections between the processing device 140 and theapparatus 110, the terminal 130, and/or the storage device 150. Theconnection may be a wired connection, a wireless connection, any othercommunication connection that can enable data transmission and/orreception, and/or any combination of these connections. The wiredconnection may include, for example, an electrical cable, an opticalcable, a telephone wire, or the like, or any combination thereof. Thewireless connection may include, for example, a Bluetooth™ link, aWi-Fi™ link, a WiMax™ link, a WLAN link, a ZigBee link, a mobile networklink (e.g., 3G, 4G, 5G, etc.), or the like, or a combination thereof. Insome embodiments, the communication port 240 may be and/or include astandardized communication port, such as RS232, RS485, etc. In someembodiments, the communication port 240 may be a specially designedcommunication port. For example, the communication port 240 may bedesigned in accordance with the digital imaging and communications inmedicine (DICOM) protocol.

FIG. 3 is a schematic diagram illustrating exemplary hardware and/orsoftware components of an exemplary mobile device on which the terminal130 may be implemented according to some embodiments of the presentdisclosure. As illustrated in FIG. 3, the mobile device 300 may includea communication platform 310, a display 320, a graphics processing unit(GPU) 330, a central processing unit (CPU) 340, an I/O 350, a memory360, and a storage 390. In some embodiments, any other suitablecomponent, including but not limited to a system bus or a controller(not shown), may also be included in the mobile device 300. In someembodiments, a mobile operating system 370 (e.g., iOS™, Android™,Windows Phone™, etc.) and one or more applications 380 may be loadedinto the memory 360 from the storage 390 in order to be executed by theCPU 340. The applications 380 may include a browser or any othersuitable mobile apps for receiving and rendering information relating toimage processing or other information from the processing device 140.User interactions with the information stream may be achieved via theI/O 350 and provided to the processing device 140 and/or othercomponents of the system 100 via the network 120.

To implement various modules, units, and their functionalities describedin the present disclosure, computer hardware platforms may be used asthe hardware platform(s) for one or more of the elements describedherein. A computer with user interface elements may be used to implementa personal computer (PC) or any other type of work station or terminaldevice. A computer may also act as a server if appropriately programmed.

FIG. 4 is a block diagram illustrating an exemplary processing device400 according to some embodiments of the present disclosure. Theprocessing device 400 may be an example of the processing device 140illustrated in FIG. 1. At least a portion of the processing device 400may be implemented on the computing device 200 (e.g., the processor 210)illustrated in FIG. 2 or the mobile device 300 as illustrated in FIG. 3.The processing device 400 may include an image acquisition module 410, aray casting module 420, and an image generation module 430.

The image acquisition module 410 may be configured to obtain a firstvolume image and a second volume image. The first volume image mayinclude a plurality of first voxels, and the second volume image mayinclude a plurality of second voxels. Some of the plurality of firstvoxels and the some of the plurality of second voxels may correspond toa same object (e.g., an organ). A voxel (e.g., a first voxel, a secondvoxel) may have at least one voxel value. The at least one voxel valueof a voxel may include a color value and/or a transparency value torepresent the color and/or the transparency of the voxel, respectively.

The ray casting module 420 may be configured to cast a plurality of raysthrough the first volume image and/or the second volume image. A ray maycorrespond to a pixel of a fused image. The ray casting module 420 mayalso be configured to obtain fusion data for a ray. At least a portionof the plurality of rays may pass through the volume of the first volumeimage and/or the volume of the second volume image.

The image generation module 430 may be configured to generate a fusedimage based on the fusion data. For each pixel of the fused image, theimage generation module 430 may determine a corresponding pixel valuebased on the fusion data. Further, based on the plurality of pixels andthe corresponding pixels values, the image generation module 430 maygenerate a fused image.

It is noted that the above descriptions about the processing device 400are only for illustration purposes, and not intended to limit thepresent disclosure. It is understood that after learning the majorconcept and the mechanism of the present disclosure, a person ofordinary skill in the art may alter the processing device 400 in anuncreative manner. The alteration may include combining and/or splittingmodules or units, adding or removing optional modules or units, etc. Allsuch modifications are within the protection scope of the presentdisclosure.

FIG. 5 is a block diagram illustrating an exemplary ray casting module500 according to some embodiments of the present disclosure. The raycasting module 500 may be an example of the ray casting module 420illustrated in FIG. 4. The ray casting module 500 may be implemented onthe computing device 200 (e.g., the processor 210) illustrated in FIG. 2or the mobile device 300 as illustrated in FIG. 3. The ray castingmodule 500 may include a sampling position setting unit 510, a referenceposition selection unit 520, a fusion data determination unit 530, and apixel value determination unit 540.

The sampling position setting unit 510 may be configured to set a seriesof sampling positions (or sampling points) along a ray. The series ofsampling positions of a ray may be determined from the origin of the rayor from the point (entering point) where the ray enters into the volumeof the first/second volume image. The sampling position setting unit 510may determine multiple sampling positions along the ray until the end ofthe ray, to the point (exiting point) where the ray exits the volume, oruntil a certain count of sampling positions have been determined.

The reference position selection unit 520 may be configured to select,based on voxel values of the first voxels along the ray, a referenceposition from the series of sampling positions. In some embodiment, thereference position selection unit 520 may first determine a referencevoxel, and then designate the sampling position corresponding to thereference voxel as the reference position.

The fusion data determination unit 530 may be configured to determine,based on a voxel value of each of at least one first voxel correspondingto the reference position and voxel values of second voxelscorresponding to at least some of the series of sampling positions,fusion data of the ray. In some embodiments, the voxel value of eachfirst or second voxel may include one or more sub-values. In someembodiments, the voxel value may be represented as a vector includingone or more elements. For instance, for an RGBA color space, the voxelvalue may be represented as a vector including three elementscorresponding to the color channels Red, Green, and Blue.

The pixel value determination unit 540 may be configured to determine,based at least in part on the fusion data of the ray, a pixel value of apixel of the fused image that corresponds to the ray. In someembodiments, the pixel value determination unit 540 may determine thepixel value of the pixel of the fused image based on a volume renderingtechnique. Exemplary volume rendering techniques may include aray-casting algorithm, a shear-warp algorithm, a frequency domainalgorithm, a splatting algorithm, or the like, or any combinationthereof. In some embodiments, based on the determined pixels and aplurality preset parameters, the pixel value determination unit 540 maygenerate the fused image. In some embodiments, the fused image may bedisplayed on a display device (e.g., the display 320).

It is noted that the above descriptions about the processing device 400and the ray casting module 500 are only for illustration purposes, andnot intended to limit the present disclosure. It is understood thatafter learning the major concept and the mechanism of the presentdisclosure, a person of ordinary skill in the art may alter theprocessing device 400 and/or the ray casting module 500 in an uncreativemanner. The alteration may include combining and/or splitting modules orunits, adding or removing optional modules or units, etc. All suchmodifications are within the protection scope of the present disclosure.

FIG. 6 is a flowchart illustrating an exemplary process 600 forgenerating an image to be displayed according to some embodiments of thepresent disclosure. The process 600 may be implemented by the processingdevice 400 illustrated in FIG. 4. In some embodiments, the process 600illustrated in FIG. 6 may be implemented in the image processing system100 illustrated in FIG. 1 (e.g., by the processing device 140). Forexample, the process 600 illustrated in FIG. 6 may be stored in astorage device (e.g., the storage device 150, the storage 220, a ROM, aRAM) in the form of instructions, and invoked and/or executed by one ormore processors (e.g., the processor 210) of the processing device 140.

In 610, the processing device 140 (e.g., the image acquisition module410) may obtain a first volume image and a second volume image. Thefirst volume image may include a plurality of first voxels, and thesecond volume image may include a plurality of second voxels. Some ofthe plurality of first voxels and the some of the plurality of secondvoxels may correspond to a same object (e.g., an organ, a lesion). Eachvoxel (e.g., a first voxel, a second voxel) may include at least onevoxel value. The at least one voxel value of a voxel may include a colorvalue and/or a transparency value to represent the color and/or thetransparency of the voxel, respectively. For example, the first volumeimage and/or the second volume image may adopt an RGBA color space, andthe at least one voxel value may include four voxel values representingthe four channels: Red (R), Green (G), Blue (B), and Alpha (A).

In some embodiments, the first volume image and the second volume imagemay be three-dimensional (3D) images and/or 3D data set representing the3D images. The 3D data set may represent a scalar field acquired by acertain imaging scanner. Exemplary imaging scanners may include an X-rayscanner, a computed tomography (CT) scanner, a micro-CT scanner, aposition emission tomography (PET) scanner, a magnetic resonance imaging(MRI) scanner, an ultrasound scanner, a bone densitometry scanner, orthe like, or any combination thereof.

In some embodiments, the 3D images may be a reconstructed 3D image, suchas a 3D PET image, a 3D CT image, a 3D MR image, a 3D ultrasound image,or the like, or a combination thereof. The 3D images may be generatedbased on a 3D reformation technique. Exemplary 3D reformation techniquesmay include a volume rendering (VR) technique, a maximum intensityprojection (MIP) technique, a minimum intensity projection (MinIP)technique, an average intensity projection (AIP) technique, a surfaceshade display (SSD) technique, a virtual endoscopy (VE) technique, orthe like, or any combination thereof. In some embodiments, the firstvolume image may be a 3D PET image generated based on the maximumintensity projection (MIP) technique, and the second volume image may bea 3D CT/MR image generated based on the volume rendering (VR) technique.

In some embodiments, the first volume image and the second volume imagemay be obtained from a same source, e.g., using a same multi-modalityscanner to scan a same object (such as a patient) concurrently orsequentially. The first volume image and the second volume image mayinclude image regions corresponding to the same object. In such asituation, the image spaces of the first volume image and the secondvolume image may both correspond to the scan space of the multi-modalityimaging system. By performing an image matching operation with respectto the coordination systems of the first volume image and the secondimage, such as a translation operation, a rotation operation, and/or ascaling operation, the first volume image may match the second volumeimage, so as to establish a correspondence between first voxels andsecond voxels that represent same positions or elements (such as tissue,organs, tumors, etc., of a patient). In some embodiments, the firstvolume image and the second volume image may be generated directly withsuch a correspondence, and therefore the matching operation may beomitted. For example, the same coordinates in the coordinate system(e.g., a Cartesian coordinate system, a digital imaging andcommunications in medicine (DICOM) coordinate system) of the firstvolume image and the second volume image may directly represent the sameposition in the scan space of the multi-modality imaging system. In someembodiments, a patient may be scanned by a PET-CT imaging systemincluding a PET scanner and a CT scanner. The first volume image may bea 3D PET image. The second volume image may be a 3D CT image.

In some embodiments, the first volume image and the second volume imagemay be obtained from different sources, e.g., using differentsingle-modality scanners to scan a same object (or different objects(e.g., a first region of a patient scanned by one single-modalityscanner only partially overlapping a second region of the patientscanned by another single-modality scanner). The first volume image andthe second volume image may include image regions corresponding to thesame portion of the object (such as a patient). In such a situation, animage registration operation may be performed to establish acorrespondence between first voxels and second voxels that representsame positions or elements (such as tissues, organs, tumors, etc., of apatient). For example, a patient may be subject to a PET scan by asingle-modality PET scanner in a PET scanning room and a CT scan by asingle-modality CT scanner in a CT scanning room. The first volume imagemay be a 3D PET image. The second volume image may be a 3D CT image.However, the 3D PET image and 3D CT image may correspond to differentscan spaces, i.e., the scan space of the single-modality PET scanner andthe scan space of the single-modality CT scanner. In such a situation,an image registration operation may be performed on the 3D CT image andthe 3D PET image. Merely as an example, the processor may outline afirst contoured surface of the patient in the 3D PET image and a secondcontoured surface of the patient in the 3D CT image. Through the imageregistration, the first contoured surface in the 3D PET image may matchthe second contoured surface of the patient in the 3D CT image, and thefirst volume image and the second volume image may be positioned intothe same coordinate system in which the corresponding portions overlapwith each other.

In some embodiments, after the image registration operation or the imagematching operation, the first volume image and the second volume imagemay at least partially overlap. For example, image portions (or voxels)of the first volume image and the second volume image representing asame object (e.g., an organ) may overlap after the image registrationoperation or the image matching operation.

In 620, the processing device 140 (e.g., the ray casting module 420) maycast a plurality of rays through the first volume image and/or thesecond volume image. A ray may correspond to a pixel of a fused image.In some embodiments, the plurality of rays may be parallel.Alternatively, the plurality of rays may be non-parallel (e.g.,convergent) to achieve a desired effect (e.g., a desired display effect)of the fused image. Due to the location and shape of the first volumeimage and/or the second volume image, the location of the source of theplurality of rays (e.g., where the plurality of rays start), thedirections of the plurality of rays, etc., one or more of the pluralityof rays may be outside the volume of the first volume image and/or thesecond volume image. The pixels corresponding to such rays may form thebackground of the fused image.

In 630, the processing device 140 (e.g., the ray casting module 420) mayobtain fusion data for each of at least a portion of the plurality ofrays. The at least a portion of the plurality of rays may pass throughthe volume of the first volume image and/or the volume of the secondvolume image. In some embodiments, along at least a portion of the ray(e.g., the portion that lies within the volume of the first volume imageand/or the volume of the second volume image) or along the whole ray,sampling positions (or sampling points) may be determined along the ray.In some embodiments, the determined sampling positions may beequidistant. A sampling position within the volume of the first volumeimage may correspond to one or more voxels. A sampling position withinthe volume of the second volume image may correspond to one or morevoxels. For example, the first volume image and the second volume imagemay partially overlap. A sampling position inside the overlappingportion may correspond to one or more first voxels and one or moresecond voxels at the same time. A sampling position outside theoverlapping portion may correspond to one or more first voxels when thesampling position is inside the first volume image, or correspond to oneor more second voxels when the sampling position is inside the secondvolume image. In some embodiments, a sampling position may be locatedinside a first voxel or a second voxel. In some embodiments, a samplingposition may be located between/among one or more first voxels and/orsecond voxels.

To obtain the fusion data of a ray, a sampling value may be obtained ateach of at least some of the sampling positions in the first volumeimage and/or the second volume image. For example, a first sample valuemay be obtained at a sampling position based on one or more first voxelscorresponding to the sampling position. As another example, a secondsample value may be obtained at a sampling position based on one or moresecond voxels corresponding to the sampling position. The sampling valuemay be a single value or a vector including multiple elements. When thefirst/second volume image adopts a color space including multiple colorchannels (the alpha channel representing the transparency is not treatedas a color channel herein), the sampling value may be a vector includingelements corresponding to the multiple color channels. For example, foran RGBA color space, the sample value may be a vector including threeelements corresponding to the color channels R, G, and B. The fusiondata corresponding to the ray may be obtained based on the obtainedsampling values.

In some embodiments, the plurality of first/second voxels may be treatedas cubes densely packed to form the volume of the first/second volumeimage. Accordingly, the ray may be viewed as passing through the cubesof the first/second volume image, and each sampling position inside thevolume of the first/second volume image may be located inside afirst/second voxel. To obtain a first/second sampling value of such asampling position, the ray casting module 420 may determine afirst/second voxel inside which the sampling position is located, anddetermine the first/second sampling value based on one or more voxelvalues of the determined first/second voxel. For example, thefirst/second sampling value may be a vector formed by the voxel valuesof the first/second voxel corresponding to different color channels.

In some embodiments, the plurality of first/second voxels may be treatedas points. In such a situation, a sampling position inside the volume ofthe first/second volume image may be located between/among differentvoxels. To obtain a first/second sampling value of such a samplingposition, the ray casting module 420 may determine one or morefirst/second voxels adjacent to the sampling position, and determine thefirst/second sampling value based on voxel values of the one or morefirst/second voxels using an interpolation or extrapolation approach.For example, the first/second sampling value may be determined using atrilinear interpolation approach to determine the first/second sampleddata based on the voxel values of the one or more first/second voxels.

In some embodiments, one or more of the plurality of sampling positionsmay be determined outside the volume of the first volume image or thesecond volume image. Taking a sampling position outside the volume ofthe second volume image for example, such a sampling position maycorrespond to no second voxel. Correspondingly, a sampling value (secondsampling value) corresponding to that sampling position with respect tothe second volume image may be zero. Similarly, for a sampling positionoutside the volume of the first volume image, a corresponding samplingvalue (first sampling value) with respect to the first volume image mayalso be zero.

In some embodiments, for a ray passing through the volume of the secondvolume image, the ray casting module 420 may determine a referenceposition and a reference voxel value corresponding to the referenceposition. The reference position and the reference voxel value may bedetermined based on voxel values of the first voxels included in thefirst volume image.

In some embodiments, for a ray passing through the volume of the secondvolume image, the ray casting module 420 may determine the correspondingpixel value using an alpha blending technique.

In some embodiments, for a ray, voxels of the first volume image andvoxels of the second volume image that are along the ray may beselectively used for obtaining the corresponding fusion data. Moredescriptions of the obtaining of the fusion data for each of theplurality of rays may be found elsewhere in the present disclosure. See,e.g., FIG. 7 and the descriptions thereof.

In 640, the processing device (e.g., the image generation module 430)may generate a fused image based on the fusion data. In someembodiments, for a pixel of the fused image, the image generation module430 may determine a corresponding pixel value based on the fusion data.Further, based on the plurality of pixels and the corresponding pixelsvalues, the image generation module 430 may generate the fused image.More descriptions of generating the image to be displayed based on thefusion data may be found elsewhere in the present disclosure. See, e.g.,FIG. 7 and the descriptions thereof.

It should be noted that the above description of the process 600 ismerely provided for the purposes of illustration, and not intended tolimit the scope of the present disclosure. For persons having ordinaryskills in the art, multiple variations or modifications may be madeunder the teachings of the present disclosure. However, those variationsand modifications do not depart from the scope of the presentdisclosure. For example, the process 600 may further include anoperation for pre-processing (e.g., denoising) the first volume imageand the second volume image. As another example, in 620, the processingdevice 140 may use a volume rendering technique other than MIP to obtainthe fusion data.

FIG. 7 is a flowchart illustrating an exemplary process 700 fordetermining fusion data of a ray through the first volume image and/orthe second volume image according to some embodiments of the presentdisclosure. The process 700 may be implemented by the ray casting module500 illustrated in FIG. 5. In some embodiments, the process 700illustrated in FIG. 7 may be implemented in the image processing system100 illustrated in FIG. 1 (e.g., by the processing device 140). Forexample, the process 700 illustrated in FIG. 7 may be stored in astorage device (e.g., the storage device 150, the storage 220, a ROM, aRAM) in the form of instructions, and invoked and/or executed by one ormore processors (e.g., the processor 210) of the processing device 140.

The process 700 may be performed on each of the plurality of rays.

In 710, the processing device 140 (e.g., the sampling position settingunit 510) may set a series of sampling positions (or sampling points)along a ray. The series of sampling positions may be determined from theorigin of the ray or from the point (entering point) where the rayenters into the volume of the first/second volume image. The samplingposition setting unit 510 may determine the sampling positions along theray until the end of the ray, to the point (exiting point) where the rayexits the volume, or until a specific count of sampling positions havebeen determined.

In some embodiments, the sampling position setting unit 510 maydetermine a sampling interval for the ray. Based on the samplinginterval, the series of sampling positions may be equidistantlydetermined.

In some embodiments, the series of sampling positions may include astart sampling position and an end sampling position. The start samplingposition may correspond to the origin of the ray, or the entering pointof the ray where the ray enters the volume of the second volume image.The end sampling position may correspond to the end point of the ray, orthe exiting point of the ray where the ray exits the volume of thesecond volume image, or a dynamically determined position where acertain condition is satisfied. For example, the sampling positionsetting unit 510 may sequentially sample at least one transparency valueat each of the series of sampling positions from the start samplingposition, and monitor an accumulation of the already obtainedtransparency values. When the accumulation of the already obtainedtransparency values reaches a certain threshold (e.g., one), the currentsampling position may be dynamically determined as the end samplingposition.

In 720, the processing device 140 (e.g., the reference positionselection unit 520) may select, based on voxel values of the firstvoxels along the ray, a reference position from the series of samplingpositions.

In some embodiments, the reference position selection unit 520 may firstdetermine a reference voxel. The reference voxel may be the first voxelthat has a highest voxel value among the first voxels along the ray. Forexample, the first voxels included in the first volume image may bedetermined based on a maximum intensity projection (MIP) technique. Foreach of the first voxels along the ray, it may have an intensity value.The reference position selection unit 520 may determine a voxel with thehighest intensity value among the first voxels along the ray as thereference voxel. In some embodiments, the reference position selectionunit 520 may determine a voxel adjacent to (e.g., one or more voxelbefore or after) the voxel with the highest intensity value among thefirst voxels along the ray as the reference voxel. In some embodiments,the reference position selection unit 520 may determine an intensitythreshold. In some embodiments, the reference position selection unit520 may determine a voxel along the ray having an intensity valuegreater than the intensity threshold that appears first along the ray asthe reference voxel.

The reference position selection unit 520 may designate the samplingposition corresponding to the reference voxel as the reference position.In some embodiments, when the voxels are treated as densely packedcubes, the reference position may be the sampling position inside thereference voxel or where the reference voxel resides. In someembodiments, when the voxels are treated as points, the referenceposition may be determined based on the position of the reference voxel.For example, the reference position may be the sampling position, amongthe sampling positions, closest to the reference voxel.

In 730, the processing device 140 (e.g., the fusion data determinationunit 530) may determine, based on a voxel value of each of at least onefirst voxel corresponding to the reference position and voxel values ofsecond voxels corresponding to at least some of the series of samplingpositions, fusion data of the ray.

In some embodiments, the voxel value of each first or second voxel mayinclude one or more sub-values. For example, the voxel value may includeone or more sub-values representing different colors (e.g., red, green,blue, etc.) of the voxel. The voxel value may also include a sub-valuerepresenting the transparency of the voxel. In some embodiments, thevoxel value may be represented as a vector including one or moreelements. An element may represent a factor of the voxel, such as color,luminance, transparency, or the like, or any combination thereof.

In some embodiments, at the reference position, the fusion datadetermination unit 530 may obtain a first sampling value based on thevoxel value of each of the at least one first voxel corresponding to thereference position. The fusion data determination unit 530 may alsoobtain, at each sampling position from the start sampling position tothe end sampling position, a second sampling value based on the voxelvalue of each of at least one second voxel corresponding to the samplingposition. As a result, a plurality of second sampling values may beobtained. Then based on the at least one first sampling value and theplurality of second sampling values, the fusion data of the ray may bedetermined.

In some embodiments, as described above, the reference position may bedetermined based on the MIP technique. When a voxel is treated as acube, the first/second sampling value may be the voxel value of thefirst/second voxel inside which the corresponding sampling position islocated. When a voxel is treated as a point, the first/second samplingvalue may be determined based on the voxel values of one or morefirst/second voxels adjacent to the corresponding sampling position. Insome embodiments, the fusion data determination unit 530 may designate amean of the voxel values of the one or more first/second voxels adjacentto the corresponding sampling position as the first/second samplingvalue. In some embodiments, the fusion data determination unit 530 mayuse an interpolation or extrapolation approach (e.g., a trilinearinterpolation approach) to obtain the first/second sampling value basedon the voxel values of the first/second voxels adjacent to thecorresponding sampling position, respectively.

In some embodiments, the fusion data determination unit 530 may firstdesignate the second sampling value of the start sampling position as afirst preliminary value. And then the fusion data determination unit 530may obtain a second preliminary value by updating, for each samplingposition from the sampling position next to the start sampling positionto the reference position and based at least in part on the secondsampling value of the sampling position, the first preliminary value.Further, the fusion data determination unit 530 may obtain a thirdpreliminary value by updating the second preliminary value based on thefirst sampling value. At last, the fusion data determination unit 530may obtain the fusion data of the ray by updating, for each samplingposition from the sampling position next to the reference position tothe end sampling position and based at least in part on the secondsampling value of the sampling position, the third preliminary value.

In some embodiments, the obtaining of the second preliminary value byupdating, for each sampling position from the sampling position next tothe start sampling position to the reference position and based at leastin part on the second sampling value of the sampling position, the firstpreliminary value includes combining a current first preliminary valueand the second sampling value using an alpha blending technique. Thesecond preliminary value may include a second preliminary color valueand a second preliminary transparency value. For each sampling positionfrom the sampling position next to the start sampling position to thereference position, the second preliminary value including the secondpreliminary color value and the second preliminary transparency valuemay be determined according to Equations (1) and (2):C _(i) ^(Δ)=(1−A _(i−1) ^(Δ))C _(i) +C _(i−1) ^(Δ),  (1)andA _(i) ^(Δ)=(1−A _(i−1) ^(Δ))A _(i) +A _(i−1) ^(Δ),  (2)where C_(i) is the color value of the ith sampling position along theray, A_(i) is the opacity value of the ith sampling position along theray, (1−A_(i)) is the transparency value of the ith sampling positionalong the ray, C_(i) ^(Δ) is the second preliminary color valuerepresenting an accumulation of the color value from the first samplingposition to the ith sampling position along the ray, and (1−A_(i) ^(Δ))is the second preliminary transparency value representing anaccumulation of the transparency value from the first sampling positionto the ith sampling position along the ray.

In some embodiments, the obtaining of the third preliminary value byupdating the second preliminary value based on the first sampling valueincludes combining the second preliminary value and the first samplingvalue using the alpha blending technique. The first sampling value mayinclude a first color sampling value and a first transparency samplingvalue. The third preliminary value may include a third preliminary colorvalue and the third preliminary transparency value which may bedetermined according to Equations (3) and (4):C _(i+1) ^(Δ)=(1−A′ _(i))C′ _(i) +C _(i) ^(Δ),  (3)andA _(i+1) ^(Δ)=(1−A _(i) ^(Δ))A′ _(i) +A _(i) ^(Δ),  (4)where C′_(i) is the first color sampling value at the reference samplingposition, (1−A′_(i)) is the first transparency sampling value at thereference sampling position, C_(i) ^(Δ) is the second preliminary colorvalue, C_(i+1) ^(Δ) is the third preliminary color value, A_(i) ^(Δ) isthe second preliminary transparency value, and A_(i+1) ^(Δ) is the thirdpreliminary transparency value.

In some embodiments, the fusion data determination unit 530 maydetermine the third preliminary value as a weighted sum of the secondpreliminary value and the first sampling value. The weights of thesecond preliminary value and the first sampling value may bepredetermined. For example, the third preliminary value may include athird preliminary color value and the third preliminary transparencyvalue which may be determined according to Equations (5) and (6):C _(i+1) ^(Δ) =M ₁ C′ _(i) +N ₁ C _(i) ^(Δ),  (5)andA _(i+1) ^(Δ) =M ₂ A′ _(i) +N ₂ A _(i) ^(Δ),  (6)where C′_(i) is the first color sampling value at the reference sampleposition, C_(i) ^(Δ) is the second preliminary color value, C_(i+1) ^(Δ)is the third preliminary color value, A′_(i) is the first transparencysampling value at reference sample position, A_(i) ^(Δ) is the secondpreliminary transparency value, (1−A_(i+1) ^(Δ)) is the thirdpreliminary transparency value, and M₁, N₁, M₂, and N₂ are predeterminedweights of the first color sampling value, the second preliminary colorvalue, first transparency sampling value and the second preliminarytransparency value, respectively.

In some embodiments, the obtaining of the fusion data of the ray byupdating, for each sampling position from the sampling position next tothe reference position to the end sampling position and based at leastin part on the second sampling value of the sampling position, the thirdpreliminary value may include combining a current third preliminaryvalue and the second sampling value using the alpha compositingtechnique. The fusion data of the ray may include a fusion color valueand a fusion transparency value of the ray. The determination of thefusion color value and a fusion transparency value may be similar tothat of the second preliminary color value and the second preliminarytransparency value.

In some embodiments, for each updating operation, a terminationcondition may be considered. The termination condition may be at leastone of the sampling position being outside the volume of the secondvolume image, or the accumulation of the transparency value reaching acertain threshold (e.g., one), or the sampling position corresponding tothe end point of the ray. Once the termination condition is satisfied,the updating process may terminate. For example, if the accumulation ofthe transparency value reaches a certain threshold (e.g., one) at asampling position before the reference position, then the operation ofobtaining the third preliminary value by updating the second preliminaryvalue based on the first sampling value may be omitted. As anotherexample, if the sampling position corresponding to the exiting point ofthe ray out of the volume of the second volume image occurs before thereference position, then the operation of obtaining the thirdpreliminary value by updating the second preliminary value based on thefirst sampling value may be omitted.

In 740, the processing device 140 (e.g., the pixel value determinationunit 540) may determine, based at least in part on the fusion data ofthe ray, a pixel value of a pixel of the fused image that corresponds tothe ray.

In some embodiments, the pixel value determination unit 540 maydetermine the pixel value of the pixel of the image to be displayedbased on a volume rendering technique. Exemplary volume renderingtechniques may include a ray-casting algorithm, a shear-warp algorithm,a frequency domain algorithm, a splatting algorithm, or the like, or anycombination thereof.

In some embodiments, based on the determined pixels and a plurality ofparameters (e.g., preset parameters), the pixel value determination unit540 may generate the image. In some embodiments, the generated image maybe displayed on a display device (e.g., the display 320). The pluralityof parameters may include image type, image size, image resolution,window width, window level, or the like, or any combination thereof. Insome embodiments, the pixel value determination unit 540 may perform awindow width adjustment and/or a window level adjustment to adjust thedisplay effect of the image. For example, the displayed image may be agrayscale image of a human body, the pixel value determination unit 540may perform the window width adjustment to limit a pixel value range ofthe pixels displayed in the grayscale image, and may perform the windowlevel adjustment to limit the central value of the pixel value range.Thus, the grayscale image may show more details of different tissues ofthe human body.

It should be noted that the above description of the process 700 ismerely provided for the purposes of illustration, and not intended tolimit the scope of the present disclosure. For persons having ordinaryskills in the art, multiple variations or modifications may be madeunder the teachings of the present disclosure. However, those variationsand modifications do not depart from the scope of the presentdisclosure. For example, in 710, the series of sampling positions alongthe ray may be determined based on the start sampling position, the endsampling position, and a certain count of sampling positions. As anotherexample, in 720, the processing device 140 may use a volume renderingtechnique other than MIP (such as, a minimum intensity projection(MinIP) technique, an average intensity projection (AIP) technique,etc.) to determine the reference position.

FIG. 8 is a flowchart illustrating an exemplary process 800 for imagefusion according to some embodiments of the present disclosure. Theprocess 800 may be implemented by the ray casting module 500 illustratedin FIG. 5. In some embodiments, the process 800 illustrated in FIG. 8may be implemented in the image processing system 100 illustrated inFIG. 1 (e.g., by the processing device 140). For example, the process800 illustrated in FIG. 8 may be stored in a storage device (e.g., thestorage device 150, the storage 220, a ROM, a RAM) in the form ofinstructions, and invoked and/or executed by one or more processors(e.g., the processor 210) of the processing device 140.

In some embodiments, the operations in process 800 may correspond to orbe included in operations that have be described in FIG. 6 and FIG. 7.For example, the operation 802 may correspond to operation 610 in FIG.6. As another example, the connect of operations 804 to 812 may be foundelsewhere in the present disclosure (e.g., FIG. 7 and the descriptionsthereof).

In 802, the processing device 140 (e.g., the image acquisition module410) may obtain a first volume image and a second volume image. Thefirst volume image and the second volume image may be generated bydifferent medical imaging devices. The different medical imaging devicesmay be preset to scan a same part (e.g., an organ) of a same subject(e.g., a patient). Also, the first volume image and the second volumeimage corresponding to the same part of the same subject may be obtainedfrom any storage device.

In 804, the processing device 140 (e.g., the ray casting module 420) mayobtain, according to a result of performing an integration for a ray inthe first volume image, a position with the maximum gray value(corresponding to the reference position mentioned above) along the rayin the first volume image and a first sampling color of the position(corresponding to the first sampling value of the reference positionmentioned above).

In 806, the processing device 140 (e.g., the ray casting module 420) maydetermine a second sampling color (corresponding to the secondpreliminary value mentioned above) by sampling in the second volumeimage along the ray until a position corresponding to the position withthe maximum gray value in the first volume image (also referred to as“the corresponding position in the second volume image”).

In 808, the processing device 140 (e.g., the ray casting module 420) maydetermine a third sampling color (corresponding to the third preliminaryvalue mentioned above) based on the first sampling color and the secondsampling color. In some embodiments, the ray casting module 420 maydetermine the third sampling color by fusing the first sampling colorand the second sampling color.

Specifically, the processing device 140 may position the first volumeimage and the second volume image to be in a stereo coordinate system.Since a planar image may be formed by connecting a plurality of squarepixels, a volume image may be composed of cube-shaped voxels. For a raypassing through a volume image along a path, it may sequentially passthrough multiple voxels on the path. Each of the voxels may have its owncolor.

Based on the above description, the processing device 140 may perform anintegration for each of the plurality of rays until the voxel with themaximum gray value in the first volume image, and determine the positionof the voxel with the maximum gray value and the first sampling color ofthe voxel. Further, the processing device 140 may sample in the secondvolume image until a position which is same as the position of the voxelwith the maximum gray value in the first volume image (also referred toas “the corresponding position in the second volume image”), anddetermine the second sampling color of the voxel at the correspondingposition in the second volume image. The processing device 140 may thengenerate a third sampling color by fusing the first sampling color andthe second sampling color when each pair of voxels in the first volumeimage and the second volume image is fusing. As used herein, a pair ofvoxels in the first volume image and the second volume image may referto a first voxel in the first volume image and a second voxel in thesecond volume image with the same position as the first voxel in thefirst volume image.

In 810, the processing device 140 (e.g., the ray casting module 420) maydetermine the corresponding position in the second volume image as astarting point, and sample along the ray that the corresponding positionin the second volume image belongs to until the end point of the ray toobtain a sampling result.

In 812, the processing device 140 (e.g., the ray casting module 420) mayfuse the sampling result with the third sampling color to obtain afourth sampling color (corresponding to the fusion data mentioned above)corresponding to a fused image.

Then, the processing device 140 (e.g., the ray casting module 420) maysample each of the voxels after the position with the maximum gray valuealong the ray in the second volume image one by one. The processingdevice 140 may fuse the sampling results sequentially with the thirdsampling color to obtain the fourth sampling color.

That is to say, for any ray passing a same path in the first volumeimage and the second volume image, the voxels from the position of thestart point to the position (corresponding to the reference position)with maximum gray value along the ray may be fused one by one, which canincrease the similarity between the color change in the fused image andthat of the unfused image(s). The superposition of the color values ofthe voxels from the position with the maximum gray value may furtherincrease the fineness of image fusion. It is convenient to maintain acorrect imaging depth on the basis of fusion, and avoid the situationthat the image content is unclear or occluded due to the fusion ofvoxels in dark positions of the first volume image and the voxels in thelight position of the second volume image. It is also convenient toclearly reflect the depth of the first volume image and/or the secondvolume image after the fusion process, which may ensure that a correctpositional relationship between the tissues in the first volume imageand tissues in the second volume image can be correctly reflected duringvarious interactive operations, and the accuracy of the fused image canbe improved.

In some embodiments, the first volume image and the second volume imagemay be a maximum intensity projection image and a volume renderingimage, respectively. The maximum intensity projection image may be animage generated based on medical imaging perspective and renderingtechnologies, which may display a projection of the internal tissuestructure of the scanned object. The volume rendering image may be animage obtained by continuously superposing color values of voxels duringintegrating rays.

In related art, if the maximum intensity projection image and the volumerendering image, for example, images of a whole body, are directlyfused, the heart, the tumor, and the brain tissue may be obviouslydisplayed in front and shield other portions of the body, the layeringmay be not clear, and the depth relationship cannot be reflectedcorrectly. By using the technical solution in the present disclosure,after the fusion process, the structure may be clear and portions of thebody may not shield each other, and the depth may be correctlyreflected, which is convenient for observation and diagnosis.

As another example, if the tumor is under the ribs, directly fusing themaximum intensity projection image and the volume rendering image maycause the tumor to be shielded. By using the technical solution in thepresent disclosure, after fusing the maximum intensity projection imageand the volume rendering image, the depth relationship between the tumorand the ribs may be correct, and the tumor under the ribs may bedirectly observed.

It can be seen that the technical solution in the present disclosure canbe used for the fusion of a maximum intensity projection image of PETand a volume rendering image of CT. It can guarantee that a correctpositional relationship between the tissues of the maximum intensityprojection image and the volume rendering image may be always correctlyreflected during various interaction operations.

In some embodiments, the first volume image and the second volume imagemay a volume rendering image and a maximum intensity projection image,respectively.

It should be understood that in an actual scenario, the first volumeimage and the second volume image may also be any other type of volumeimages other than the volume rendering image and the maximum intensityprojection image.

In some embodiments, in 804, the processing device 140 may determine themaximum gray value along the ray of the first volume image according tothe result of integrating the ray of the first volume image. Further,the processing device 140 may determine the position corresponding tothe maximum gray value along the ray in the first volume image and thecorresponding first sampling color.

The determining the maximum gray value along the ray of the first volumeimage according to the result of integrating the ray of the first volumeimage may include: traversing all voxels on the ray; determining a voxelcorresponding to the maximum gray value according to a result of thetraversing. The position and the sampling color of the voxel may be theposition where the maximum gray value is located in the first volumeimage and the corresponding first sampling color.

The color of each voxel in a volume image may be composed of threeprimary colors of Red (R), Green (G), and Blue (B). According to thecomparison of the values of the colors, the voxel with the maximum grayvalue may be selected as a base point for fusion among all the voxelsalong the ray. Each voxel may have a corresponding position and acorresponding color.

In 810, the processing device 140 may take the corresponding position asthe starting point, and sample in the second volume image in a samplingstep along the ray. The fusing the sampling result with the thirdsampling color to obtain the fourth sampling color corresponding to afused image, may include fusing sampling results of each sampling stepone by one with the third sampling color to obtain the fourth samplingcolor.

After the operation 808, the sampling color of each voxel from the firstvoxels next to the position with the maximum gray value along the raymay be determined. The processing device 140 may fuse the samplingcolors one by one with the third sampling color to obtain the finalfourth sampling color.

Each of the sampling steps may include one or more voxels. If thepositions of the first volume image and the second volume image are notcorresponding one-to-one, there may be cases of fusing a plurality ofvoxels of the first volume image and a single voxel of the second volumeimage, or fusing a single voxel of the first volume image and aplurality of voxels of the second volume image, to meet the actualdiagnostic requirement.

FIG. 9 is a flowchart illustrating another exemplary process 900 forimage fusion according to some embodiments of the present disclosure.

In 910, in the process for integrating a ray, the processing device 140(e.g., the ray casting module 420) may first calculate the maximum grayvalue (denoted as MaxGray), the corresponding position (denoted asMaxStep), and the sampling color (denoted as SampleColor) of the maximumintensity projection.

In 920, for the volume rendering image, the processing device 140 (e.g.,the ray casting module 420) may sample from the start point to MaxStepto obtain a sampling color (denoted as vecSampleColor) which may befused with the SampleColor of the maximum intensity projection. Thefusion result may be further assigned to vecSampleColor. The fusionmethod may be determined according to the following Equations (7) and(8):vecSampleColor⁺=(1.0-vecSampleColor.a)*sampleColorMIP.rgb  (7)vecSampleColor.a ⁺=sampleColorMIP.a*(1.0-vecSampleColor.a)  (8)

In some embodiments, the Equations (7) and (8) may be another expressionform of the Equations (1) and (2), wherein (1.0-vecSampleColor.a)represents the transparency value in the second volume image,sampleColorMIP.rgb represents the sampling color value corresponding tothe reference position in the first volume image, sampleColorMIP.arepresents the transparency value corresponding to the referenceposition in the first volume image.

In 930, for the volume rendering image, the processing device 140 (e.g.,the ray casting module 420) may sample from the position correspondingto the first voxel next to the MaxStep (denoted as MaxStep+1) to theending position of the ray (denoted as EndStep). The results of eachsampling steps may be fused with vecSampleColor, and the fusion resultmay be assigned as vecSampleColor. That is to say, after the MaxGray,the voxels may be sampled one by one. The fusion method for eachsampling step may be determined according to Equations:vecSampleColor.rgb⁺=(1.0-vecSampleColor.a)*vecGlobalColor.rgb*vecGlobalColor.a  (9)andvecSampleColor.a ⁺=vecGlobalColor.a*(1.0-vecSampleColor.a)  (10)

In some embodiments, the Equations (9) and (10) may be anotherexpression form of the Equations (3) and (4), wherein(1.0-vecSampleColor.a) represents the transparency value in the secondvolume image, vecGlobalColor.rgb represents the sampling color valuescorresponding to the position after the MaxGray in the second volumeimage, vecGlobalColor.a represents the transparency values correspondingto the position after the MaxGray in the second volume image.

When the volume rendering image is fused with the maximum densityintensity projection image of volume data, the maximum gray value andthe position corresponding to the maximum gray value of the maximumintensity projection on each of the rays may be first determined duringthe ray casting process. Then, an accumulation along the ray may beperformed on the volume rendering image. When the integration of each ofthe rays reaches the position with the maximum gray value in the maximumintensity projection, the sampling color at this time may be fused withthe sampling color of the maximum intensity projection. Further, theintegration of the ray may be continued until the end. The final displayresults may reflect the depth relationship between the volume datacorrectly.

FIG. 10 is a block diagram illustrating an exemplary image fusion system1000 according to some embodiments of the present disclosure. As shownin FIG. 10, the image fusion system 1000 may include a first obtainingmodule 1010, a second obtaining module 1020, a first sampling module1030, a first fusion module 1040, a second sampling module 1050 and asecond sampling module 1060. The first obtaining module 1010 may beconfigured to obtain the first volume image and the second volume image.The second obtaining module 1020 may be configured to obtain, accordingto a result of integrating a ray in the first volume image, a positionwith the maximum gray value of the first volume image and a firstsampling color thereof. The first sampling module 1030 may be configuredto determine the second sampling color by sampling along the ray in thesecond volume image to a position corresponding to the position with themaximum gray value in the first volume image. The first fusion module1040 may be configured to determine a third sampling color based on thefirst sampling color and the second sampling color. The second samplingmodule 1050 may be configured to determine the corresponding position inthe second volume image as a starting point, and sample the secondvolume image to the end point of the ray to obtain a sampling result.The second sampling module 1060 may be configured to fuse the samplingresult with the third sampling color to obtain a fourth sampling colorcorresponding to the fused image

The image fusion system 1000 may use the solutions described in any ofthe embodiments shown in FIGS. 1 to 9. Therefore, it has all thetechnical effects described above, and the details are not describedherein again.

In some embodiments, the first volume image and the second volume imagemay be a MIP image and a volume rendering image, respectively.

In some embodiments, the second obtaining module 1020 may be configuredto determine the maximum gray value of the first volume image accordingto the result of integrating a ray in the first volume image, anddetermine the position with the maximum gray value along the ray in thefirst volume image and the corresponding first sampling color.

In some embodiments, optionally, the second obtaining module 1020 may beconfigured to traverse all voxels along the ray and determine the voxelcorresponding to the maximum gray value according to a result of thetraversing. The position and the sampling color of the voxel may be theposition where the maximum gray value is located in the first volumeimage and the corresponding first sampling color.

In some embodiments, optionally, the second sampling module 1050 may beconfigured to take the corresponding position as the starting point, andsample in a sampling step along the ray in the second volume image. Thefusing of the sampling result with the third sampling color to obtainthe fourth sampling color corresponding to the fused image may includefusing sampling results of each sampling step one by one with the thirdsampling color.

In some embodiments, optionally, each sampling step may include morethan zero voxel.

FIG. 11 is a block diagram illustrating an exemplary medical equipment1100 according to some embodiments of the present disclosure. As shownin FIG. 11, the medical equipment 1100 include the image fusion system1000 described in FIG. 10. Therefore, the medical equipment 1100 mayrealize the same technical effects as the image fusion system 1000described in FIG. 10, and whose details are not repeated here.

In addition, according to some embodiments of the present disclosure, animage fusion terminal is provided. The image fusion terminal may includea processor and a memory. The memory may be configured to storeinstructions. When the instructions are executed by the processor, theimage fusion terminal may be caused to implement the method as describedin the embodiments in FIG. 8.

FIG. 12a and FIG. 12b are fusion images generated based on volumerendering and MIP techniques according to some embodiments of thepresent disclosure. In FIG. 12a , a first CT image was determined usinga volume rendering technique and a second PET image using an MIPtechnique. The first CT image and the second PET image may include imageregions corresponding to a same patient. For each of the pixels in thefirst CT image in the image regions corresponding to the same patient,it may have a corresponding pixel in the second PET image. Pixels in thefirst CT image and the corresponding pixels in the second PET image weredirectly fused to obtain the pixel values of fused pixels and furthergenerated the fusion image, which is shown in the FIG. 12a . As aresult, the organs (e.g., heart), the tissue (e.g., brain tissue),and/or a lesion (e.g., a tumor) of the patient may be exposed outsidethe body, which does not reflect a correct depth relationship. In FIG.12b , a fusion image was generated using the fusion techniques describedin the present disclosure. A pixel of the fusion image may be determinedbased on the CT image volume data and the PET volume data. As a result,the organs (e.g., heart), the tissue (e.g., brain tissue) and/or alesion (e.g., a tumor) may be obscured by bones of the patient, whichreflect the correct depth relationship.

FIG. 13a and FIG. 13b are fusion images with different window widthsand/or window levels according to some embodiments of the presentdisclosure. As shown in the in FIG. 13a , the tumor was observed throughthe gap between the ribs of the patient. While in FIG. 13b , the tumorwas blocked due to the adjustment of the window width and/or the windowlevel on the fusion data. In other words, the adjustment of window widthand/or the window level may relate to a field of view (FOV) and thedepth of an observation. A proper adjustment of window width and/or thewindow level may lead to a desired observation.

FIG. 14 is a fusion image according to some embodiments of the presentdisclosure. The fusion image was generated by fusing a 3D MR scalpanatomical image and a 3D MR time of flight (TOF) angiography image. Thescalp anatomical image was a 3D MR image generated using a volumerendering technique. The TOF angiography image was a 3D MR imagegenerated using an MIP technique. As shown in the fusion image in FIG.14, the nerves and the vessels are wrapped in the brain tissue, whichreflects the correct depth relationship.

Having thus described the basic concepts, it may be rather apparent tothose skilled in the art after reading this detailed disclosure that theforegoing detailed disclosure may be intended to be presented by way ofexample only and may be not limiting. Various alterations, improvements,and modifications may occur and are intended to those skilled in theart, though not expressly stated herein. These alterations,improvements, and modifications are intended to be suggested by thisdisclosure, and are within the spirit and scope of the exemplaryembodiments of this disclosure.

Moreover, certain terminology has been used to describe embodiments ofthe present disclosure. For example, the terms “one embodiment,” “anembodiment,” and/or “some embodiments” mean that a particular feature,structure or characteristic described in connection with the embodimentmay be included in at least one embodiment of the present disclosure.Therefore, it may be emphasized and should be appreciated that two ormore references to “an embodiment” or “one embodiment” or “analternative embodiment” in various portions of this specification arenot necessarily all referring to the same embodiment. Furthermore, theparticular features, structures or characteristics may be combined assuitable in one or more embodiments of the present disclosure.

Further, it will be appreciated by one skilled in the art, aspects ofthe present disclosure may be illustrated and described herein in any ofa number of patentable classes or context including any new and usefulprocess, machine, manufacture, or composition of matter, or any new anduseful improvement thereof. Accordingly, aspects of the presentdisclosure may be implemented entirely hardware, entirely software(including firmware, resident software, micro-code, etc.) or combiningsoftware and hardware implementation that may all generally be referredto herein as a “unit,” “module,” or “system.” Furthermore, aspects ofthe present disclosure may take the form of a computer program productembodied in one or more computer readable media having computer readableprogram code embodied thereon.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including electro-magnetic, optical, or thelike, or any suitable combination thereof. A computer readable signalmedium may be any computer readable medium that may be not a computerreadable storage medium and that may communicate, propagate, ortransport a program for use by or in connection with an instructionexecution system, apparatus, or device. Program code embodied on acomputer readable signal medium may be transmitted using any appropriatemedium, including wireless, wireline, optical fiber cable, RF, or thelike, or any suitable combination of the foregoing.

Computer program code for carrying out operations for aspects of thepresent disclosure may be written in any combination of one or moreprogramming languages, including an object to be recognized orientedprogramming language such as Java, Scala, Smalltalk, Eiffel, JADE,Emerald, C++, C#, VB. NET, Python or the like, conventional proceduralprogramming languages, such as the “C” programming language, VisualBasic, Fortran 2103, Perl, COBOL 2102, PHP, ABAP, dynamic programminglanguages such as Python, Ruby, and Groovy, or other programminglanguages. The program code may execute entirely on the user's computer,partly on the user's computer, as a stand-alone software package, partlyon the user's computer and partly on a remote computer or entirely onthe remote computer or server. In the latter scenario, the remotecomputer may be connected to the user's computer through any type ofnetwork, including a local part network (LAN) or a wide part network(WAN), or the connection may be made to an external computer (forexample, through the Internet using an Internet Service Provider) or ina cloud computing environment or offered as a service such as a Softwareas a Service (SaaS).

Furthermore, the recited order of processing elements or sequences, orthe use of numbers, letters, or other designations therefore, may be notintended to limit the claimed processes and methods to any order exceptas may be specified in the claims. Although the above disclosurediscusses through various examples what may be currently considered tobe a variety of useful embodiments of the disclosure, it may be to beunderstood that such detail may be solely for that purposes, and thatthe appended claims are not limited to the disclosed embodiments, but,on the contrary, are intended to cover modifications and equivalentarrangements that are within the spirit and scope of the disclosedembodiments. For example, although the implementation of variouscomponents described above may be embodied in a hardware device, it mayalso be implemented as a software only solution, for example, aninstallation on an existing server or mobile device.

Similarly, it should be appreciated that in the foregoing description ofembodiments of the present disclosure, various features are sometimesgrouped together in a single embodiment, figure, or description thereoffor the purposes of streamlining the disclosure aiding in theunderstanding of one or more of the various inventive embodiments. Thismethod of disclosure, however, may be not to be interpreted asreflecting an intention that the claimed subject matter requires morefeatures than are expressly recited in each claim. Rather, inventiveembodiments lie in less than all features of a single foregoingdisclosed embodiment.

In some embodiments, the numbers expressing quantities or propertiesused to describe and claim certain embodiments of the application are tobe understood as being modified in some instances by the term “about,”“approximate,” or “substantially.” For example, “about,” “approximate,”or “substantially” may indicate ±20% variation of the value itdescribes, unless otherwise stated. Accordingly, in some embodiments,the numerical parameters set forth in the written description andattached claims are approximations that may vary depending upon thedesired properties sought to be obtained by a particular embodiment. Insome embodiments, the numerical parameters should be construed in lightof the number of reported significant digits and by applying ordinaryrounding techniques. Notwithstanding that the numerical ranges andparameters setting forth the broad scope of some embodiments of theapplication are approximations, the numerical values set forth in thespecific examples are reported as precisely as practicable.

Each of the patents, patent applications, publications of patentapplications, and other material, such as articles, books,specifications, publications, documents, things, and/or the like,referenced herein may be hereby incorporated herein by this reference inits entirety for all purposes, excepting any prosecution file historyassociated with same, any of same that may be inconsistent with or inconflict with the present document, or any of same that may have alimiting affect as to the broadest scope of the claims now or laterassociated with the present document. By way of example, should there beany inconsistency or conflict between the description, definition,and/or the use of a term associated with any of the incorporatedmaterial and that associated with the present document, the description,definition, and/or the use of the term in the present document shallprevail.

In closing, it is to be understood that the embodiments of theapplication disclosed herein are illustrative of the principles of theembodiments of the application. Other modifications that may be employedmay be within the scope of the application. Thus, by way of example, butnot of limitation, alternative configurations of the embodiments of theapplication may be utilized in accordance with the teachings herein.Accordingly, embodiments of the present application are not limited tothat precisely as shown and describe.

We claim:
 1. A method, implemented on a machine including at least oneprocessor and at least one storage device, the method comprising:obtaining a first volume image and a second volume image, the firstvolume image including a plurality of first voxels, the second volumeimage including a plurality of second voxels; casting a plurality ofrays through at least one of the first volume image or the second volumeimage, wherein each of the plurality of rays corresponds to a pixel ofan image to be displayed; for each of at least a portion of theplurality of rays: setting a series of sampling positions along the ray;selecting, based on voxel values of the first voxels along the ray, areference position from the series of sampling positions; determining,based on a voxel value of at least one first voxel corresponding to thereference position and voxel values of second voxels corresponding to atleast some of the series of sampling positions, fusion data of the ray;and determining, based at least in part on the fusion data of the ray, apixel value of a pixel of the image to be displayed that corresponds tothe ray.
 2. The method of claim 1, wherein the selecting; based on voxelvalues of the first voxels along the ray, a reference position from theseries of sampling positions includes: determining a reference voxel,wherein the reference voxel is the first voxel that has a highest voxelvalue among the first voxels along the ray; and designating the samplingposition corresponding to the reference voxel as the reference position.3. The method of claim 1, wherein: the series of sampling positionsinclude a start sampling position and an end sampling position; and thedetermining, based on a voxel value of at least one first voxelcorresponding to the reference position and voxel values of secondvoxels corresponding to at least some of the series of samplingpositions; fusion data of the ray includes: determining, based on thevoxel value of each of the at least one first voxel corresponding to thereference position and the voxel values of the second voxelscorresponding to sampling positions from the start sampling position tothe end sampling position, the fusion data of the ray.
 4. The method ofclaim 3; where in the determining, based on the voxel value of each ofthe at least one first voxel corresponding to the reference position andthe voxel values of the second voxels corresponding to samplingpositions from the start sampling position to the end sampling position,the fusion data of the ray includes: obtaining, at the referenceposition, a first sampling value based on the voxel value of the atleast one first voxel corresponding to the reference position;obtaining, at each sampling position from the start sampling position tothe end sampling position, a second sampling value based on the voxelvalue of each of at least one second voxel corresponding to the samplingposition; and determining; based on the first sampling value and theplurality of second sampling values, the fusion data of the ray.
 5. Themethod of claim 4, wherein: the first sampling value is obtained byperforming an interpolation or extrapolation on the voxel value of eachof the at least one first voxel corresponding to the reference position;and the second sampling value is obtained by performing an interpolationor extrapolation on the voxel value of each of the at east one secondvoxel corresponding to the sampling position.
 6. The method of claim 4,where in the determining, based on the first sampling value and theplurality of second sampling values, the fusion data of the rayincludes: designating the second sampling value of the start samplingposition as a first preliminary value; obtaining a second preliminaryvalue by updating the first preliminary value, wherein the updating thefirst preliminary value is performed for each sampling position from thesampling position next to the start sampling position to the referenceposition and based at least in part on the second sampling value of thesampling position; obtaining a third preliminary value by updating thesecond preliminary value based on the first sampling value; andobtaining the fusion data of the ray by updating the third preliminaryvalue, wherein the updating the third preliminary value is performed foreach sampling position from the sampling position next to the referenceposition to the end sampling position and based at least in part on thesecond sampling value of the sampling position.
 7. The method of claim6, wherein the obtaining a second preliminary value by updating thefirst preliminary value includes combining a current first preliminaryvalue and the second sampling value using an alpha blending technique;the obtaining a third preliminary value by updating the secondpreliminary value includes combining the second preliminary value andthe first sampling value using the alpha blending technique; or theobtaining the fusion data of the ray by updating the third preliminaryvalue includes combining a current third preliminary value and thesecond sampling value using the alpha blending technique.
 8. The methodof claim 1, wherein the obtaining a first volume image and a secondvolume image includes: performing an image registration between thefirst volume image and the second volume image, wherein the first volumeimage and the second volume image include image regions corresponding toa same object.
 9. The method of claim 1, wherein the determining, basedat least in part on the fusion data of the ray, a pixel value of a pixelof the image to be displayed that corresponds to the ray is based on avolume rendering technique.
 10. The method of claim 9, wherein thedetermining, based at least in part on the fusion data of the ray, apixel value of a pixel of the image to be displayed that corresponds tothe ray includes: performing a window width adjustment or a window leveladjustment on the fusion data of the ray.
 11. The method of claim 1,wherein: the first volume image is obtained via a maximum intensityprojection (MIP) technique.
 12. The method of claim 1, wherein: thefirst volume image is a positron emission computed tomography (PET)image; and the second volume image is a computed tomography (CT) imageor a magnetic resonance (MR) image.
 13. A system, comprising: at leastone storage medium including a set of instructions; at least oneprocessor in communication with the at least one storage medium, whereinwhen executing the set of instructions, the at least one processor isdirected to cause the system to perform operations including: obtaininga first volume image and a second volume image, the first volume imageincluding a plurality of first voxels, the second volume image includinga plurality of second voxels; casting a plurality of rays through atleast one of the first volume image or the second volume image, whereineach of the plurality of rays corresponds to a pixel of an image to bedisplayed; for each of at least a portion of the plurality of rays:setting a series of sampling positions along the ray; selecting, basedon voxel values of the first voxels along the ray, a reference positionfrom the series of sampling positions; determining, based on a voxelvalue of at least one first voxel corresponding to the referenceposition and voxel values of second voxels corresponding to at leastsome of the series of sampling positions, fusion data of the ray; anddetermining, based at least in part on the fusion data of the ray, apixel value of a pixel of the image to be displayed that corresponds tothe ray.
 14. The system of claim 13, wherein to select, based on voxelvalues of the first voxels along the ray, a reference position from theseries of sampling positions, the at least one processor is directed tocause the system to perform additional operations including: determininga reference voxel, wherein the reference voxel is the first voxel thathas a highest voxel value among the first voxels along the ray; anddesignating the sampling position corresponding to the reference voxelas the reference position.
 15. The system of claim 13, wherein theseries of sampling positions include a start sampling position and anend sampling position; and to determine, based on a voxel value of atleast one first voxel corresponding to the reference position and voxelvalues of second voxels corresponding to at least some of the series ofsampling positions, fusion data of the ray, the at least one processoris directed to cause the system to perform additional operationsincluding: determining; based on the voxel value of each of the at leastone first voxel corresponding to the reference position and the voxelvalues of the second voxels corresponding to sampling positions from thestart sampling position to the end sampling position, the fusion data ofthe ray.
 16. The system of claim 13, wherein to determine, based on thevoxel value of each of the at least one first voxel corresponding to thereference position and the voxel values of the second voxelscorresponding to sampling positions from the start sampling position tothe end sampling position, the fusion data of the ray, the at least oneprocessor is directed to cause the system to perform additionaloperations including: obtaining, at the reference position, a firstsampling value based on the voxel value of the at least one first voxelcorresponding to the reference position; obtaining, at each samplingposition from the start sampling position to the end sampling position,a second sampling value based on the voxel value of each of at least onesecond voxel corresponding to the sampling position, thereby obtaining aplurality of second sampling values; and determining, based on the firstsampling value and the plurality of second sampling values, the fusiondata of the ray.
 17. The system of claim 16, wherein: the first samplingvalue is obtained by performing an interpolation or extrapolation on thevoxel value of each of the at least one first voxel corresponding to thereference position; and the second sampling value is obtained byperforming an interpolation or extrapolation on the voxel value of eachof the at least one second voxel corresponding to the sampling position.18. The system of claim 17, wherein to determine, based on the firstsampling value and the plurality of second sampling values, the fusiondata of the ray, the at least one processor is directed to cause thesystem to perform additional operations including: designating thesecond sampling value of the start sampling position as a firstpreliminary value; obtaining a second preliminary value by updating thefirst preliminary value, wherein the updating the first preliminaryvalue is performed for each sampling position from the sampling positionnext to the start sampling position to the reference position and basedat least in part on the second sampling value of the sampling position;obtaining a third preliminary value by updating the second preliminaryvalue based on the first sampling value; and obtaining the fusion dataof the ray by updating the third preliminary value, wherein the updatingthe third preliminary value is performed for each sampling position fromthe sampling position next to the reference position to the end samplingposition and based at least in part on the second sampling value of thesampling position.
 19. The system of claim 13, wherein the first volumeimage is obtained via a maximum intensity projection (MIP) technique.20. A non-transitory computer readable medium storing instructions, theinstructions, when executed by at least one processor, causing the atleast one processor to implement a method comprising: obtaining a firstvolume image and a second volume image, the first volume image includinga plurality of first voxels, the second volume image including aplurality of second voxels; casting a plurality of rays through at leastone of the first volume image or the second volume image, wherein eachof the plurality of rays corresponds to a pixel of an image to bedisplayed; for each of at least a portion of the plurality of rays:setting a series of sampling positions along the ray; selecting, basedon voxel values of the first voxels along the ray, a reference positionfrom the series of sampling positions; determining, based on a voxelvalue of at least one first voxel corresponding to the referenceposition and voxel values of second voxels corresponding to at leastsome of the series of sampling positions, fusion data of the ray; anddetermining, based at least in part on the fusion data of the ray, apixel value of a pixel of the image to be displayed that corresponds tothe ray.